-
Notifications
You must be signed in to change notification settings - Fork 224
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
backend start script: check ulimit and increase soft limit if possible #1234
Conversation
I think this code would be a good place to document why we need to increase the limit. |
I added a docstring to the |
src/start_fact_backend.py
Outdated
""" | ||
Each process has a hard limit and a soft limit for the maximum number of files opened at the same time. Since | ||
FACT makes extensive use of multiprocessing features, it uses up a lot of those file descriptors and if we run | ||
out, this raises an OSError. To mitigate this, we try to increase the soft limit and print a warning if the | ||
hard limit is low. With the default configuration, FACT uses about 560 file descriptors (and potentially many | ||
more if you crank up the worker counts). | ||
""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can specify which multiprocessing features use file descriptors?
While I know that its queues, I would not expect the average reader to think of queues when thinking about multiprocessing.
Also, can you document the calculation that lead to 560 descriptors by default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I extended the docstring quite a bit and added information on how the FDs are distributed
No description provided.